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Abstract 


The main idea of multi-frame super resolution (SR) algorithms 
is to recover a single high-resolution image from a sequence of low 
resolution ones of the same object. The success of the SR approaches 
is often related to a well registration and restoration steps. Therefore, 
we propose a new approach based on fluid image registration and we 
use a second order partial differential equation (PDE) to treat both 
the registration and restoration steps that guarantees the success of SR 
algorithms. Since the registration step is usually a variational ill-posed 
model, a mathematical study is needed to check the existence of the 
solution to the regularized problem. Thus, we prove the existence and 
uniqueness of the well posed fluid image registration and assure also 
the existence of the used second order PDE in the restoration step. 
The results show that the proposed method is competitive with the 
existing methods. 
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1 Introduction 


Image super resolution represents an active research area in the imagery 
domain. The main principle of this method is to recover a high-resolution 
(HR) image through a single image (Single-image SR) [38, 35, 19], or using 
a series of low-resolution (LR) ones that are down-sampled, ill-registered, 
blurred and usually noised (multi-frame SR) [42, 24, 11]. SR intervenes in 
numerous domains, such as video surveillance and medical diagnostics [16]. 

The multi-frame and single SR are both intensively studied in order 
to improve the resolution of an image. In single SR technique, the missing 
information in the LR image is estimated from a large number of training. 
Then it success depends on the availability of a convenient dictionary which 
is not ensured in large application areas. While for the multi-frame SR 
the success is related to the approximation accuracy of the motion between 
the LR images. Since multiframe SR is more complicated in the presence 
of different degradation operators, Sina Farsiu et al [13] proposed the fast 
robust super resolution (RSR). The aim of this new technique is to break 
down the SR method into two steps: looking for a degraded HR image from 
the estimated LR frames after a registration procedure; then in a second 
step, restoring it [22]. 

On another side, the Euler-Lagrange equation associated to nonlinear 
PDEs have also been treated in the super-resolution context with great results. 
One of the consistent nonlinear PDE was proposed by Maiseli et al. [28], 
which takes the advantages of perona-Malik equation and the TV norm. 
This adaptive diffusion-based PDE can efficiently preserve image features 
but it suffers from the blurring effect. A more robust PDE was proposed 
by El Mourabit et al. [11] which take into consideration the coherence- 
enhancing property and avoids blur much better. However, when the blur 
and noise levels are too high, the obtained HR image still contain some 
artefacts, in particular the blur. More recently a fourth-order nonlinear PDE 
is then introduced with much more constrained diffusion that stops the edges 
destruction and preserve the smooth areas and texture [23]. In contrast, the 
blur apparition problem still a remaining weakness of this PDE. 

The first step of multiframe SR aim is to guest the motion between 
the LR frames to register all LR images in a common way [44]. This stage 
is the key of the success of multiframe SR algorithms and without a good 
estimation of the motion between the LR sequence, the super-resolution 
becomes very limited [5]. 

Afterwards, the selection of a regularization function in the deblurring 


An Enhanced Fluid Registration for 
Image Multi-Frame Super Resolution 257 


and denoising process (the last step of SR technique) is very important to 
avoid different artefacts. Indeed, great care must be made in the choice of 
this function. There is numerous of super-resolution approaches that rely on 
regularization framework [17, 12, 29]. Even if these methods give promising 
results they have some shortcomings. One of them is the staircasing effect 
and the blur apparition in the flat regions. Although the noise and blur, in 
smooth regions, can be reduced by adjusting the regularization parameter, 
the texture information is however blurred. 

Furthermore, deep learning has been widely used to address the single 
image super-resolution task. One of the famous work is the one proposed 
by Dong et al. [9], where the authors introduce the CNN-based SR method 
(SRCNN), which is done in three convolutional steps. The three steps are 
realized such as: patch extraction, non-linear mapping, and reconstruction. 
The SRCNN has shown attractive results, however, the need of large HR 
images for data training still an ambarrassing problem. Then, Dong et al. 
introduced an accelerated version of SRCNN which called FSRCNN [8], 
which aims is to incorporate the upsampling operation into the network. As 
a result, the restoration quality is efficient in a remarkable execution time. 
In contrast, this approach remains relatively shallow. Other approaches 
are then elaborated to fix the issues of the previous approach, see [18] 
for more details. Recently, Zhang et al. [43] used more advances on deep 
learning to propose the similar network (DnCNN), which shows great success. 
More recently, Chen et al.[6] investigated a deep convolutional network- 
based SR (CISRDCNN) framework for compressed images, which gives 
promising results. 

In this paper, we follow the steps of the RSR method, while the registra- 
tion approach and the second step are improved. In fact, we are developed 
two things: 


* We propose a modified fluid image registration in a well-posed func- 
tional framework independent of time to handle the registration part 
of the super-resolution part. 


x To ensure this choice, we show the existence and the uniqueness of the 
solution using classical mathematics approaches [1, 20]. 


* Since the problems of deblurring and denoising are in most cases ill- 
posed, in the second step of SR, we introduce a fourth order PDE to 
remove noise and blur. 
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x The two modified steps help to increase the robustness of the of the 
proposed SR algorithm. This is justified in the numerical part. 


This paper follows this configuration: Section 2 introduces the super resolu- 
tion formulation. We introduce then the fluid registration and how to define 
the deformations operators in Section 3. In Section 4, we describe briefly 
the proposed PDE and the main algorithm used in the deconvolution and 
denoising step. Finally, simulated and real results with comparisons of the 
proposed SR method to some available methods are presented in Section 5. 


2 Problem Formulation 


In the presence of many degradation factors, the acquired images are always 
in a non desired resolution. In fact, the obtained images are decimated, 
noised and also blurred. We consider that the LR images are taken in 
the same conditions with one sensor. The link between the HR image X 
(described by a vector of size [r2N? x 1] where r is the enhancement factor) 
and the associated low resolution frames Y; (represented by a vector of size 
[N? x 1]), is given by 


Y, = DFZHX + ex Vk =1,2,...,n, (1) 
where: 


H: the convolution operator describing the blur of size [r?N? x r?.N?]. 


D: the decimation operator which is related to the desired resolution of 
the HR image of size [N? x r?N?]. 


F,: are the motions or the warp matrix of size [r?N? x r?.N?], representing 
randomize transformation between the LR frames. 
ex: is the additive Gaussian noise vector of size [N? x 1]. 


The aim of multi-frame super resolution is the recovery of an ideal 
HR image X. Due to the complexity of the problem we split it in two 
procedure [13]: 


1. Approximating the transformation matrix F; between each couple of 
LR images and merge the HR image B with noise and blur. 


2. Computing the HR image X through the blurring and noisy one B. 


We start by the approximation of the warp matrix F; and also by ensuring 
the existence of the solution in a suitable functional space [30, 24, 3]. 
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3 The Construction of the Warp Matrix Fi, 


We obtain the warp matrix F; for each frame through a fluid registration 
algorithm, after a transition from discrete to continuous images using 2- 
linear interpolation. Let us denote by Y;(a) the intensity of the kth image 
at coordinate « € 2 C R?, where © is defined as the domain of the image. 
The image registration problem is formulated as 


Y(7) = Ypona)) fork=2,....n and Veen. (2) 
Our goal is to find the velocity deformation v; between the images, where 
Up = Opug + Vug.vp, (3) 


ux are the deformations between each frame. Unfortunately, this problem 
is ill-posed. We have therefore to choose an appropriate regularization 
operator S. Since we know the success of the fluid registration to handle 
different problems in image registration [27], we propose to use it in a 
well-posed functional framework. The image fluid registration problem is 
defined as 


min J(uz,), where J(uz) = D(Y, Yz, ux) + BS(vz), 


such that vp, = uz, + Vup.vp, 


where 7 denotes the set of admissible transformations, and @ the regularisa- 
tion parameter. 
D : is the squared difference measure defined by Lebesgue L? norm as 


DY. Ye.) = f (Yelun(e)) - ¥(e))* ae, (5) 
Q 
and the fluid regularisation is defined as 
2,A 2 
S(vp) = | ju (traceV)* + 5 trace(V da, (6) 
Q 

where V is the Cauchy strain tensor (for ||Vvz|| < 1) defined as V(vuz,) = 
(Voz + Vu,)/2, and ys and A are the Lamé parameters. To ensure the 
existence of a unique solution to the problem (4), we have to chose firstly 


a functional framework. A natural choice of the functional space is the 
Sobolev space H4(Q). To demonstrate the ellipticity of 7 and the strict 
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convexity, we use the Korn’s inequality [31]. Thus, the admissible space of 
transformation is defined as 


T = Hg (). (7) 


We use also a classical result [2] to show the lower semi-continuity (l.s.c) of the 
function 7. In the following, we present the main theorem of the existence 
and uniqueness of the solution to the proposed fluid image registration. 


Theorem 1 Let Q be a regular bounded open subset of R?, and let f be in 
I?(Q). Then, the minimization problem 


min J(ur), (8) 


UpET 


admits a unique solution. 


Proof: To demonstrate that the function / admits a unique solution, we 
use the classical steps in optimization [10], i.e. we prove that 7 is elliptic 
and weakly sequentially l.s.c. We start by the ellipticity. 


ellipticity: Using the propriety of the strain tensor in (6), we can reformu- 
late it as 


2 
rA,,. 
S(t.) = | i S> (OnjUp, + Oxjvp,)° + <(div(vg))?dz. (9) 
an jizl 2 
We need also to define the semi-norm 


1/2 


Molex ( A trace(¥ (0). (01))] 
Let up, € JT. Then 
Mone [ aac VO) + ; acl We) aa 
+ ff Wlue(0)) - ¥@))* de 


i [ acd V Oe Vedas i (Vitus = VG) ad 
> ple(vr)l6 — I fllc2@ylleall z2@); 
(10) 
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where || f||z2(a) = ||¥ Ilz-(a) = ||Yellz2(qa)- Using the Korn’s inequality, 
we can deduce that it exist a constant @ > 0 such that 


lla llan(ay S Ble(ve)|o- (11) 
By setting C = % this implies that 


T (tk) = Clini ay = WF lhcacoylleall ca): (12) 


Using the Young inequality, we have 
2 2 € 2 
T (ux) 2 Clluallin cay — CMF llz2(a) — 5lluellzn@ 


€ 
> (C — =)|luxll2acay) — CONF Zac): (13) 
5 (9) (9) 


where the parameter ¢ is chosen such as C — 5 > 0. 


Finally, we can easily check that J (ux) — oo if ||uz||71(Q) co. This 
implies that 7 is elliptic. 


Weak sequentially 1.s.c.: To demonstrate the weak sequentially l.s.c, we 
have to prove that 7 is continuous and convex. 


Continuity : 


Let (un)n>o0 be a sequence in 7 that converge to u, such as 


Un > u in L?(Q), 
and (14) 
Vun > Vu in L?(Q). 


Then there exist a subsequence (w,/),,/s9 and a function u € L?(Q) 
such as 


u,,(2) > u(x) and |u,(x)| < v1(@) ae xr € Q, (15) 
and 
Vu," («) + Vu(a) et |Vu,7(x)| < ve(x) forae xEQ. (16) 
We get 


V(u,7(@)) = 5 (Vy (x)'+Vu,7(2)) — V(u(z)) a.ex €Q. (17) 
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In addition their exist h € L'(Q), such as 


yu trace(V(u,(x))'V(u,(x)))+ 


< Ale): (18) 
© trace(V (ty (2)))? + (Fithy (@)) 
Using the Lebesgue theorem, we have 
Fh ) SF): (19) 


Strict convexity: We use a classical result of convexity based on the second 
derivative of 7. Let uz, w, € T, we have then 


wt 


JT (uz)(wk — Uk, Wk — Uk) = fu trace(V (wz — uz) ' V (we — ux) 
a ; trace(V (wr — uz))? dx (20) 
2 i trace(V (wy — up) 'V (we — upz))da. 
Using Korn’s inequality, we have 


J" (up) (we — Up, We — Up) = Cllwe— uxllin «ay (21) 
>0O for uz # Wk 


This, shows the strict convexity of 7, and concludes the proof. 


To solve the minimisation problem (8), we use the BFGS algorithm [34]. We 
finally find the warp matrix F;, using ti, (the solution of the problem (8)). 
For the fusion step, we use the algorithm in [25, 39] to compute the blurred 
and noisy HR image B = HX. After that, we carry out the last step of the 
SR algorithm, which is the restoration step. 


4 Restoration Step 


Since the problem of restoration is ill-posed, we have to be careful in the 
choice of a suitable approach for denoising and deconvolution step. The 
main purpose of this stage is to preserve image features and avoid the blocky 
effect while reducing noise and blur using high-order operator [40]. A robust 
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higher-order total variation model is proposed as an efficient solution to 
the blocky effect, called total generalised variation (TGV) [37]. Even if this 
method avoids the staircasing effect, the computational time to reach the 
solution is very significant. An alternative way is to use a combined approach 
of the first and second order regularizer proposed by Papafitsoros et al. [32] 
which eliminate noise and blur avoiding the blocky artifacts in the HR in 
less time than the TGV. Let us describe briefly the algorithm used in the 
proposed SR approach. In this paper, we use the gradient descent PDE 
with a Neumann boundary condition on 0Q, associated to the variational 
model proposed in [32]. This PDE with the initial condition X (0,7) = Xo 
(where Xo is the obtained HR image calculated by the interpolation of the LR 
image Y;) is given by 


AX =H sin(HX —B) +> ae (SS300%) 
SV 2X 29 
+(1— 7) div? (Sibeavex)), (22) 


OnX =0, on OF, 
where the divergence operator div : (IR™N°*1)2 —; R*x1, defined as 
div X.Y = X.VY, VX € (R"N?™1)2, vy E RUN?X1, (23) 


Also, the second-order divergence operator div? : (RN *x1)4 —+ RP N?x1 
with the adjointness property, is defined 


div? X.Y = X.V°Y, VX € (RUN?X1)4 ye ROANPXL, (24) 


g, and go are linear growth increasing functions defined: R —> Rt such as 
their exist four constants C1, Co, C3, Cy verifying 


g(x) <Ci(1+|2|), VeeR 
go(z) < Co(1+ |z|), VaeeR. 


Also, we assume that both g; and gg are coercive: 


gi(z) > Cs\z|, VaeR 
go(x) > Cylz|, Va ER, 


and y: the controlled regularization parameters. With this choice, we can 
reduce the staircasing compared to some previous SR methods. See for 
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a) Noisy image 


(b) HMR [15] (c) Our PDE (d) Original image 


Figure 1: The efficiency of the fourth-order PDE in reducing the staircasing 
effect for the Square image, where the noise is considered to be a Gaussian 
one with standard deviation of 0 = 25. 


example Fig. 1, where we denoise the square image using the proposed 
approach and compared to the Huber-Markov regularization (HMR) [15]. 


The existence and uniqueness of solution to this PDE is demonstrated 
using the relaxation techniques [21], based on the monotony of the operators 
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div (agyewx and div? a eaev?x ). To solve the PDE above, we 


use a Classical finite difference scheme. 


Let X;,; the discrete version of the image X, such as X;,; = X(i,j), 
t=1...M,j7=1...M, where M =rN. We give briefly the discrete version 


of the operators V and div given by 


(Vx)! ais Xiatj _ Xij if i< M 
Be 0 if i=M’ 
(Vx)? ie PCr rs _ Xj,j if j<M 
pe 0 if j=M’ 
and 
(div(p', p*))i,3 = (div(p', p”))}.; + (div(p", p”))7. 5, 
where 
Pi, — Pia, if 1<i<M 
(div(p',p”))i3 = 4 ph, if i=l 
0 if i=M 
Pi; Pe jt it 1<ape iM 
(div(p',p*))?5 = § p?, if j=1 
Pj if j=M 


(25) 


(26) 


(27) 


(28) 


(29) 


Let us define the second order discrete differential operators noted V? as 


VX = (Veeck VayX Vag X Vey X)s 


where 
Xim — 2X41 + Xi2 
VarXig = \ Xig-1 — 2Xij + Xi gq 
Xi,M-1 = 2XiM + Xi 
Xm,j — 2X1,;5 + X25 
VyyXig = § Xi-1gj — 2Xiy + Xin 


Xu-1j — 2Xuj t+ X14 


ifl1<i<M,j=l, 
ifl<i<M,1<j<M, 
ifl<i<M,j=M, 


ifi=11<j<M, 
ifl<i<M,1<j<M, 
ifi=M,1<j<M, 


(30) 


(31) 


(32) 


266 Amine Laghrib, Aissam Hadri, Moad Hakim 


and 


Aggy — Agua g —- Mega tAeajgor ilsi<M,1l<7< M, 
Aim — Xin — AG + Aaa ifl<i<M,j=M, 
Xj — X13 — Xujai t+ X1541 ifi=M,1<j<M, 
Xu,m — Xim—-Xmit X11 ifi=M,j7=M. 


VayXij = 


(33) 
In addition, for X = (X1, Xe, X3, X4) € (R™)’, we define the discrete div? 
operator as 


(div? X)ij = VaeX1,, + VyyX2,, + VayX3,, + VayX4,,, (34) 
where 
Vax = Vex; Vyy = Vyy (35) 
and 
X11 — X1,M, — XM,,1 + XMy,Mp ifi=1,j7=1, 
— Mig = Aig = Ang FA Gt ifi=11l<j<WM, 
VayXi,j = , 
Aq — Agi — Ma tA L1i<i 5 Mig HL 
AQg— Agga — Apa FA SC 1 <1 SM, 1 <7 5M. 
(36) 
To give a comprehensive form of this problem, we set the particular case 
where gi(z) = go(z) = x. As a result, the algorithm related to solve 


numerically the proposed PDE is finally given in Algorithm 1. 


5 Numerical Experiments 


In this part, our aim is to we test the ability of the elaborated algorithm in 
the SR context. Many simulated and also real results were used to test the 
performance of the proposed SR method. The first part is dedicated to the 
evaluation of the registration part while the second and third parts concern 
the main proposed SR approach. To increase the ability of the proposed 
equation to better detect edges, we choose the so-called hypersurface minimal 
function [4] defined for both the functions g; and gz by 


gi(x) = go(x) = V1 4 2?. (37) 


The input HR image X° is build by a bicubic interpolation of the LR 
reference image Y;. In the simulated experiments, we choose five examples 


An Enhanced Fluid Registration for 
Image Multi-Frame Super Resolution 267 


Algorithm 1 The proposed algorithm 

Inputs: The blurred image B; the steepest descent parameter dt. the 
regularization parameter ¥. 

To avoid the derivative singularity when X is locally constant (in the special 
case where the denominator is equal to zero); 

The procedure: 


= re z (VX)i5 
Xrtl_xn. +at(H] ,sing((HX)? ,—Bj,;))+dty div} , = —_ 
J J ( J J ) J MUVX)L PF +(V RE) 


(Vx)? 
+dty div? ; = — 
JV (VX)} )P+(VX)? 5)? +1 
VooXh; 
VJ 14 (VaeX?y)?+(V yy X25)? +2( Vey!) 


+dt (1-7) V 


VyyX 
a ie Vaux?) 24 (VyyXPj)2+2(Vay XP)? 


Vig lh” , 
(1-7) Vey = a = i,j=1,...,M 
JV 1+ (VarXp;)?+(Vyy XP)? +2( Vay Xj) 


Output: The restored HR image x 


in Fig.2, with different size and texture, to illustrate the performance of 
the proposed SR method. The used images are selected from the Berkeley 
database*, which includes 300 images with various sizes (217 x 181, 256 x 256, 
512 x 512 and 1024 x 1024) and colors. 

To construct the simulated images, we follow the degradation model (1). 
The HR image was first shifted with sub-pixel displacements to produce N 
images, then, the sequence was convoluted with a PSF and finally, zero-mean 
Gaussian noise was added to each frame of the sequence. Concerning the 
convergence of the proposed algorithm, | we end the execution at the first 
\|_X Xt ee a(n 

kan 
the performance of the proposed algorithm, the peak-signal-to-noise ratio 
(PSNR) and the structural similarity (SSIM) criterion are used. The PSNR 
is used to measure the quality of the estimated HR image, while the SSIM 


iteration n with respect to the error < 10-°. To evaluate 


“nttps://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/BSDS300/ 
html/dataset/images.html 
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(a) MRI Brain1 —_(b) MRI Brain 2 


(d) Bird (e) Flowers 


Figure 2: The original five images used in simulated tests 


is a complementary measure, which gives an indication of image quality 
based on known characteristics of the human visual system. The PSNR is 
defined by 


255? 
PSNR= 10 logig (<3) 5 


where the MSE is the mean squared error defined as 


MSE = a2) 


MN vist ayeulY (i, 5) — XG 9). 


The SSIM is calculated on multiple windows of given image, i.e. the 
measurement between two windows x and y of size N x N is defined by 


(Que by + €1)(200y + C2)(2 covgy + €3) 
(u2. + 2 + c1)(02 + 02 + €2)(Gxoy +3)’ 


SSIM(z,y) = 


where the variables, respectively, defined for x and y as follows: juz and py, 
mean; o2 and a, variance; cozy, covariance; cy = (m,L)?, cz = (m2L)? are 
two stabilizing constants; and L the dynamics of the pixel values, 255 for 
8-bit encoded image. 
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5.1 The Effectiveness of the Registration Part 


In the first experience, we construct 32 synthetic LR image from the original 
image of MRI Brain 1 such as: each frame is deformed by random vector fields, 
blurred by a Gaussian low-pass filter with size 3 x 3 and a standard deviation 
of 1.5. Then, the blurred frames are down-sampled in the two directions by 
a factor of r = 2 and a Gaussian noise was added with a standard deviation 
a = 25. In this test, we fix the deconvolution and denoising part where 
we use the proposed combined first and second order regularizer. Then, 
we compare our registration algorithm with other competitive registration 
methods in the SR context, such as: SR with probabilistic optical flow 
(POF) [14], SR with hyper-elastic (SRHE) [26], SR with deep neural network 
(Deepsum) [44], combining demons Registration (CDR) [7] and also SR with 
diffusion registration (SRDR) proposed in [25]. The obtained SR results are 
shown in Fig. 3 to see the efficiency of the proposed registration part. We 
can deduce that the proposed registration method gives a slightly better 
result compared to the other methods. For the second experiment, we 
consider the MRI Brain 1 following the same previous steps with more 
complicated random deformations between the LR images. We increase also 
the decimation factor r = 4 and also the standard deviation 0 = 35. The 
obtained HR image is depicted in Fig. 4, where comparison to other SR 
methods is done. Clearly the obtained HR image outperforms the other ones 
and the registration process is better enhanced. 

To confirm the visual comparison, we use a qualitative illustration 
through the Signal-to-noise ratio (SVR) measure defined by: 


EMM 


EML 
N M 


EML=S_S_ Xj, 


i=1 j=1 


SNR= 


N M 


EML=,|>_ >) (Yj — Xu)? 


i=1 j=1 


where Y is the clean image and X is the registered one using the denoising 
process. In fact, we present the obtained SNR values for the previous tests in 
Table 1 to show the efficiency of the proposed method in the super-resolution 
context. As expected, the proposed approaches outperforms the others in 
terms of SNR values, which validate the visual comparison. 
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(a) First LR image 


(b) 16 th LR image 


(c) 18 th LR image 


(g) SRHE [26] 


(h) SRDR [25] 


(i) Our method 


Figure 3: Comparisons of different SR methods with different registration 
procedure of the (MRI brain 1 image when the magnification factor is r = 2 
using random motion vectors) to perform the registration step. Note that 
the noise is considered with a standard deviation o = 25 


Table 1: The SNR table of the two previous tests 


Image SNR values 
POF [14] SRHE [26] CDR [7] SRDR [25] Deepsum [44] Proposed approach 
MRI Brain 1(r=2) 24.3333 26.0711 24.2222 28.3555 31.4766 37.1466 
MRI Brain 1(r=4) 22.6648 24.7688 23.1737 25.2222 32.1133 34.4399 
MRI Brain 2(r=2) 23.5555 24.6316 25.3493 28.2274 30.9333 36.3644 
MRI Brain 2(r=4) 22.5245 24.0788 24.1111 27.3373 29.2886 35.6895 
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(g) SRHE [26] (h) SRDR [25] (i) Our method 


Figure 4: Comparisons of different SR methods with different registration 
procedure of the (MRI brain 2 image when the magnification factor is r = 4 
using random motion vectors) to perform the registration step. Note that 
the noise is considered with a standard deviation o = 35. 


5.2 The Effectiveness of the Second Order Regularization Term 


To perform the restoration step of the proposed super-resolution approach, 
we fix now the registration step, which is considered as a fluid registration 
one for our method and also for the other SR approaches. We measure the 
performance of the proposed second order regularization term by considering 
a high-level of Gaussian noise, while the blur is supposed to be a Gaussian low- 
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pass filter with a size 3x3 and a standard deviation of 1.5. Indeed, we consider 
the LR frames from the Tiger original image corrupted by a Gaussian noise 
with o = 30 respectively. The obtained results, when the magnification factor 
is r = 4, using different regularization terms, such as: nonlinear fourth-order 
PDE (FPDE) [23], nonlocal-means (NLM) [33], weighted TV (SWTV) [41], 
second-order combined model (SOM) [24] and bilateral spectrum weighted 
total variation (BSWTV) [36], are illustrated in the Fig. 5. Visually, we 
can see that the proposed second order regularization preserves the image 
features better than the other terms for the three noise level. 


(d) SWTV method [41] (c) FPDE 123 (f) BSWTV [36] 


(g) The proposed SR 


Figure 5: Super resolution of the Tiger compared with different methods. 
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In the second experiment, we keep the same procedure of the SR process 
except that this time the interest is to see how the proposed regularization 
term deals with high-level of blur and noise. Indeed, we consider the Flowers 
and we augment the blur level, which is considered now as a Gaussian 
low-pass filter with a size of 7 x 7 and a standard deviation of 2 while the 
noise is considered with o = 45. Once again, as illustrated in the Fig. 6, 
the proposed regularization term outperforms the others in the quality of 
the restoration. 


(a) One LR image (b) NLM [33] _ (c) SOM [24] 
i 


(d) SWTV method [41] 


e) FPDE [23] (f) BSWTV [36] 


(g) The proposed SR 


Figure 6: Super resolution of the Flowers compared with different methods. 
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We use the same think for the third test of the Bird image, while we 
increase the blur kernel rate and o noise for the other tests. Indeed, we use 
a 5 x 5 Gaussian blur kernel and a white Gaussian noise with 0 = 30 to 
construct the sequence for the last four image. Concerning the choice of 
the parameters, the scalar weight 7 is chosen according to the better PSNR 
value for the proposed and also for the other SR methods. For instance, we 
choose y = 0.55 for the Bird image. 


(a) One LR image (b) NLM [33] 


ode Ty 


2 


(d) SWTV method [41] (c) FPDE [23] (f) BSWTV [36] 


(g) The proposed SR 


Figure 7: Super resolution of the Bird compared with different methods. 
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To ensure the success of the proposed algorithm against noise and 
misregistration errors reducing, we use the PSNR criterion in the Table 2 
and the SSIM measure in the Table 3 for the three previous tests. Knowing 
that the best score is in bold number, we can clearly show the efficiency 
of our algorithm. Also, visually, we can detect the performance of the 
proposed method in removing misregistration errors compared with the 
other methods in the smooth area, however, near strong edges, there is no 
distinct improvement compared to other methods. 


Table 2: The PSNR table of the three previous tests. The bold number 
represents the high value in each line. 


Image PSNR values 
POF [14] SRHE [26] CDR [7] SRDR [25] Deepsum [44] Proposed approach 
Tiger (r = 2) 28.88 28.22 28.17 28.66 29.26 30.47 
Tiger (r = 4) 27.92 27.62 27.59 27.84 28.94 29.89 
Flowers (r = 2) 28.18 28.07 28.58 28.97 30.06 31.16 
Flowers (r = 4) 27.68 27.78 27.37 27.22 29.33 30.29 
Bird (r = 2) 26.48 26.63 27.44 28.77 29.22 30.06 
Bird (r = 4) 25.88 25.93 26.11 26.73 28.88 29.66 


Table 3: The SSIM table of the three previous tests. The bold number 
represents the high value in each line. 


Image PSNR values 
POF [14] SRHE [26] CDR [7] SRDR [25] Deepsum [44] Proposed approach 
Tiger (r = 2) 0.8277 0.8344 0.8527 0.8499 0.8609 0.8814 
Tiger (r = 4) 0.8209 0.8184 8376 0.8333 0.8517 0.8729 
Flowers (r =2) 0.8662 0.8533 .8699 0.8733 0.8794 0.8937 
Flowers (r = 4) 0.8444 0.8527 0.8627 0.8652 0.8666 0.8399 
Bird (r = 2) 0.8008 0.8102 0.8222 0.8288 0.8366 0.8496 
Bird (r = 4) -7866 0.7933 0.7949 0.7897 0.7992 0.8109 


5.3 Real Experiments 


In the real experiments, two real data sequences are used to approve the 
proposed algorithm are presented : the Rocket and City videos. We select 
the first twenty frames in the two real data sets. The diffusion registration 
approach presented in [25] is used as the registration estimation method for 
the other methods while we use the proposed fluid registration to estimate 
the motion for our method. The reconstruction results of these sequences 
are, respectively, shown in Figs. 8 and 9, where the magnification factor is 
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selected such as: r= 2 and r = 4. From these figures, it is shown that the 
proposed approach gives a better visual effect compared to the other method. 


(a) One LR image (b) NLM [33] 


(a) SWTV method [41] (ce) FPDE [23] a (£) BSWTV [36] 


(g) The proposed SR 


Figure 8: Results on the Rocket sequence. 
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(a) One LR image 


] (f) BSWTV [36] 


(g) The proposed SR 


Figure 9: Results on the City sequence. 


6 Conclusion 


In this paper, a novel approach of the super resolution image reconstruction 
problem is introduced. We presented a fluid image registration and proved 
the existence and uniqueness of the solution using the classical functional 
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analysis. In addition, to avoid the undesirable staircasing effect during 
the registration, denoising and deconvolution steps, a fourth order PDE is 
proposed. To confirm the robustness of this approach, a set of benchmark 
image have been performed, and the investigated SR approach has proven 
its success visually and also quantitatively using two known metrics. A 
remaining question is about the treatment of other types of noise and blur, 
such as Salt& paper and Poisson noise. As a future work, we will adopt 
some learning techniques about the choice of the parameter y and also 
the regularization part to reduce the remaining blur in the restored images. 
Another interesting point is about the degrees of the efficiency of the proposed 
approach with respect to the nature of the transformations between the LR 
frames. As a future work, we will also adopt some learning techniques about 
the choice of the parameter y and also the regularization part to reduce the 
remaining blur in the restored images. 
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